inamerrata

So having found time to catch up satisfactorily on the implementation, time to get back into the blogging.

After working out what you actually want to do, the next step in implementing stuff, in my book, is to make sure you fully understand the context of the stuff you’re trying to change. In this case, the “unapproved” queue concept comes from some thoughts from quite a while ago that come under the heading “NIv2” for “new incoming v2”. For reference, the original “new incoming” was the switch to separate queues for new, byhand, and accepted packages, which was followed by a notable update shortly after to make accepted autobuilding possible.

That setup works, but has a few problems. One is that packages sit in accepted for quite a while (up to a full 24h), and occassionally in that time it can become inappropriate to actually add the package to the archive — for instance if it’s a build of a package that’s then removed from the archive. This results in an UNACCEPT, and requires manual intervention, since potentially it can result in weird inconsistencies, such as bug closures. Another is that files in the various queues aren’t tracked in the database, which can result in files (particularly upstream tarballs) being lost or left behind, and also requires some unpleasant special casing in the code. A final concern is that one of the bigger time sinks in the daily pulse is the apt-ftparchive scan, which is actually just doing work that’s already been done in order to allow packages in accepted to be autobuilt.

Put all those concerns together with the ones mentioned last week, and you get the impetus for NIv2. The principle of that is to put stuff in the queue into the database properly, simplify it, extend the queuing support to deal with different queues, and move stuff from accepted directly into the pool every quarter hour, though only pushing to buildds that frequently, leaving updates of Packages files and mirror pushes to still be daily, or every few hours or whatever.

This is what that bit looked like in the dak TODO:

[hard, long term] unchecked -> accepted should go into the db, not a suite, but similar. this would allow katie to get even faster, make madison more useful, decomplexify specialacceptedautobuild and generally be more sane. may even be helpful to have e.g. new in the DB, so that we avoid corner cases like the .orig.tar.gz disappearing ‘cos the package has been entirely removed but was still on stayofexecution when it entered new.

Obviously, it needed to get defined a little more clearly than that to work out how much the changes for the security queues I’m interested in depend on the other changes, and what they actually mean in detail. After some IRC discussion we came up with more or less the following summary of what NIv2 actually means:

How NIv2 should work.

q/accepted goes away; once packages are accepted, they get added straight into the pool and the database. At this point they’re added to a micro-suite for the buildds, and so apt-ftparchive can cache the package contents; and a symlink tree is made up so incoming.d.o works. The micro-suite might or mightn’t be a fully fledged suite in the database. Presumably it wouldn’t be visible in dists/ though. The micro-suite probably should contain packages uploaded to unstable for X hours, where X is 48 or 2x pulse-interval or similar.

This means UNACCEPT is impossible and apt-ftparchive takes less time in cron.daily.

q/byhand and q/new get new friends: q/sec-embargoed, q/sec-unembargoed, q/stable-updates. Rather than checking for “byhand” entries, or a lack of overrides; uploads get diverted to those queues if they’re uploaded to security or stable, and get moved out of there when a security team member or stable RM (ie, Joey) runs an amber/lisa-esque command.

This means sec-embargoed and sec-unembargoed need to be autobuilt.

This also means that proposed-updates can be a full suite, and point releases can be a matter of (a) adding proposed-updates to stable; (b) removing old versions from stable; (c) removing packages that have to be dropped; and users can put “stable + proposed-updates” in their sources.list reasonably sanely.

This also means that REJECTs from accepted (security) or the archive (p-u) aren’t needed.

It means the queue ends up looking like:

    unchecked      [uploads go here, insecure]

    holding        [secured]
    new            [waiting NEW processing]
    byhand         [waiting byhand processing]
    stable-updates [waiting stable RM approval]

    done           [.changes files for accepted packages]
    reject         [.changes, .reason and rejected stuff]

    bts_version_track [data for BTS]

with q/accepted disappearing and q/proposed-updates and q/old-proposed-updates being replaced by the stable-updates / oldstable-updates queues.

One of the nice parts about this is that it breaks down fairly neatly into three sets of changes: supporting additional queues, removing the accepted queue, and tracking queue stuff in the database. It does make those changes more or less dependent though — the new queues need to be autobuilt in the way accepted is at present; so it makes sense to add the queues before removing accepted so as not to lose the queue autobuild support.

The above actually can be simplified a little further — instead of managing a microsuite for changes since the last update, it’s possible to add accepted packages directly to the real suite; and generate the Packages files needed for autobuilding by querying packages accepted to unstable in the last 24 or 48 hours, simply by adding a field to track upload time for binaries as well as sources.

In any event, this thus makes generalising the queue autobuild support the next challenge.